{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入相关包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据（训练数据前10000行，测试数据前100条）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv('train_all.csv',nrows=10000)\n",
    "test_data = pd.read_csv('test_all.csv',nrows=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取全部数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_data = pd.read_csv('train_all.csv',nrows=None)\n",
    "# test_data = pd.read_csv('test_all.csv',nrows=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取训练和测试数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_columns = [col for col in train_data.columns if col not in ['user_id','label']]\n",
    "train = train_data[features_columns].values\n",
    "test = test_data[features_columns].values\n",
    "target =train_data['label'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 缺失值补全"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "处理缺失值有很多方法，最常用为以下几种：\n",
    "1. 删除。当数据量较大时，或者缺失数据占比较小时，可以使用这种方法。\n",
    "2. 填充。通用的方法是采用平均数、中位数来填充，可以适用插值或者模型预测的方法进行缺失补全。\n",
    "3. 不处理。树类模型对缺失值不明感。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 采用中值进行填充"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sklearn.preprocessing import Imputer\n",
    "# imputer = Imputer(strategy=\"median\")\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "imputer = SimpleImputer(missing_values=np.nan, strategy='mean')\n",
    "imputer = imputer.fit(train)\n",
    "train_imputer = imputer.transform(train)\n",
    "test_imputer = imputer.transform(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 特征选择概念"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在机器学习和统计学中，特征选择（英语：feature selection）也被称为变量选择、属性选择 或变量子集选择 。它是指：为了构建模型而选择相关特征（即属性、指标）子集的过程。使用特征选择技术有三个原因：\n",
    "\n",
    "    简化模型，使之更易于被研究人员或用户理解，\n",
    "    缩短训练时间，\n",
    "    改善通用性、降低过拟合（即降低方差）。\n",
    "\n",
    "要使用特征选择技术的关键假设是：训练数据包含许多冗余 或无关 的特征，因而移除这些特征并不会导致丢失信息。 冗余 或无关 特征是两个不同的概念。如果一个特征本身有用，但如果这个特征与另一个有用特征强相关，且那个特征也出现在数据中，那么这个特征可能就变得多余。\n",
    "特征选择技术与特征提取有所不同。特征提取是从原有特征的功能中创造新的特征，而特征选择则只返回原有特征中的子集。 特征选择技术的常常用于许多特征但样本（即数据点）相对较少的领域。特征选择应用的典型用例包括：解析书面文本和微阵列数据，这些场景下特征成千上万，但样本只有几十到几百个。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "def feature_selection(train, train_sel, target):\n",
    "    clf = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0, n_jobs=-1)\n",
    "    \n",
    "    scores = cross_val_score(clf, train, target, cv=5)\n",
    "    scores_sel = cross_val_score(clf, train_sel, target, cv=5)\n",
    "    \n",
    "    print(\"No Select Accuracy: %0.2f (+/- %0.2f)\" % (scores.mean(), scores.std() * 2))     \n",
    "    print(\"Features Select Accuracy: %0.2f (+/- %0.2f)\" % (scores.mean(), scores.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 删除方差较小的要素（方法一）\n",
    "VarianceThreshold是一种简单的基线特征选择方法。它会删除方差不符合某个阈值的所有要素。默认情况下，它会删除所有零方差要素，即在所有样本中具有相同值的要素。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据未特征筛选维度 (2000, 229)\n",
      "训练数据特征筛选维度后 (2000, 29)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import VarianceThreshold\n",
    "\n",
    "sel = VarianceThreshold(threshold=(.8 * (1 - .8)))\n",
    "sel = sel.fit(train)\n",
    "train_sel = sel.transform(train)\n",
    "test_sel = sel.transform(test)\n",
    "print('训练数据未特征筛选维度', train.shape)\n",
    "print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择前后区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Select Accuracy: 0.93 (+/- 0.00)\n",
      "Features Select Accuracy: 0.93 (+/- 0.00)\n"
     ]
    }
   ],
   "source": [
    "feature_selection(train, train_sel, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 单变量特征选择（方法二）\n",
    "通过基于单变量统计检验选择最佳特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据未特征筛选维度 (2000, 229)\n",
      "训练数据特征筛选维度后 (2000, 2)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "# from sklearn.feature_selection import chi2\n",
    "from sklearn.feature_selection import mutual_info_classif\n",
    "\n",
    "sel = SelectKBest(mutual_info_classif, k=2)\n",
    "sel = sel.fit(train, target)\n",
    "train_sel = sel.transform(train)\n",
    "test_sel = sel.transform(test)\n",
    "print('训练数据未特征筛选维度', train.shape)\n",
    "print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据未特征筛选维度 (2000, 229)\n",
      "训练数据特征筛选维度后 (2000, 10)\n"
     ]
    }
   ],
   "source": [
    "sel = SelectKBest(mutual_info_classif, k=10)\n",
    "sel = sel.fit(train, target)\n",
    "train_sel = sel.transform(train)\n",
    "test_sel = sel.transform(test)\n",
    "print('训练数据未特征筛选维度', train.shape)\n",
    "print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择前后区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Select Accuracy: 0.93 (+/- 0.00)\n",
      "Features Select Accuracy: 0.93 (+/- 0.00)\n"
     ]
    }
   ],
   "source": [
    "feature_selection(train, train_sel, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 递归功能消除（方法三）\n",
    "选定模型拟合，进行递归拟合，每次把评分低得特征去除，重复上诉循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False False\n",
      " False False False False False False False False False False False  True\n",
      "  True False False False False False False False False False False  True\n",
      " False  True False False False False  True False False  True False False\n",
      " False  True False  True False  True False  True False False False False\n",
      " False False False False False False False False False False False False\n",
      " False]\n",
      "[220 219 218 217 216 215 213 212 211 210 209 208 207 206 205 204 203 202\n",
      " 201 200 197 195 192 187 186 185 184 183 182 181 180 179 178 177 176 175\n",
      " 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 158 157 155\n",
      " 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 137 136\n",
      " 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 117 116 115\n",
      " 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100  99  98  97\n",
      "  95  94  93  92  91  90  89  88  87 214  86  85  84  83 189 193 199 198\n",
      " 196 194 191 190 188  81  80  79  77  74  73  72  71  69  68  67  66  65\n",
      "  64 156  63  61  60  59  58  57  55  54 138 135  50  48  46  45  44  42\n",
      "  39 119 118  38  35  31  28  25  24  22  17  15  14  96  13  12  43   1\n",
      "   1   4  82  75  78  76  26  30  70  20   7   1  62   1  51  53  49  47\n",
      "   1  27  41   1  23  21  18   1  11   1  19   1   6   1   8  29   2   5\n",
      "  10   3   9  16  32  33  34  36  37  40  52  56 159]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFECV\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=10, max_depth=2, random_state=0, n_jobs=-1)\n",
    "selector = RFECV(clf, step=1, cv=2)\n",
    "selector = selector.fit(train, target)\n",
    "print(selector.support_)\n",
    "print(selector.ranking_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用模型选择特征（方法四）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 使用LR拟合的参数进行变量选择（L2范数进行特征选择）\n",
    "LR模型采用拟合参数形式进行变量选择，筛选对回归目标影响大的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据未特征筛选维度 (2000, 229)\n",
      "训练数据特征筛选维度后 (2000, 19)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import Normalizer\n",
    "\n",
    "normalizer = Normalizer()\n",
    "normalizer = normalizer.fit(train)  \n",
    "\n",
    "train_norm = normalizer.transform(train)                            \n",
    "test_norm = normalizer.transform(test)\n",
    "\n",
    "LR = LogisticRegression(penalty='l2',C=5)\n",
    "LR = LR.fit(train_norm, target)\n",
    "model = SelectFromModel(LR, prefit=True)\n",
    "train_sel = model.transform(train)\n",
    "test_sel = model.transform(test)\n",
    "print('训练数据未特征筛选维度', train.shape)\n",
    "print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### L2范数选择参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.27519508, -0.02736226, -0.00522652,  0.90644126, -0.4310027 ,\n",
       "       -0.25110925, -0.4058899 ,  0.29059019,  0.10568508, -0.02731211])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LR.coef_[0][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择前后区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Select Accuracy: 0.93 (+/- 0.00)\n",
      "Features Select Accuracy: 0.93 (+/- 0.00)\n"
     ]
    }
   ],
   "source": [
    "feature_selection(train, train_sel, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 使用LR拟合的参数进行变量选择（L1范数进行特征选择）\n",
    "LR模型采用拟合参数形式进行变量选择，筛选对回归目标影响大的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sklearn.feature_selection import SelectFromModel\n",
    "# from sklearn.linear_model import LogisticRegression\n",
    "# from sklearn.preprocessing import Normalizer\n",
    "\n",
    "# normalizer = Normalizer()\n",
    "# normalizer = normalizer.fit(train)  \n",
    "\n",
    "# train_norm = normalizer.transform(train)                            \n",
    "# test_norm = normalizer.transform(test)\n",
    "\n",
    "# LR = LogisticRegression(penalty='l1',C=5)\n",
    "# LR = LR.fit(train_norm, target)\n",
    "# model = SelectFromModel(LR, prefit=True)\n",
    "# train_sel = model.transform(train)\n",
    "# test_sel = model.transform(test)\n",
    "# print('训练数据未特征筛选维度', train.shape)\n",
    "# print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### L1范数选择参数\n",
    "对于α的良好选择，只要满足某些特定条件，Lasso就可以仅使用少量观察来完全恢复精确的非零变量集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LR.coef_[0][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择前后区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Select Accuracy: 0.93 (+/- 0.00)\n",
      "Features Select Accuracy: 0.93 (+/- 0.00)\n"
     ]
    }
   ],
   "source": [
    "feature_selection(train, train_sel, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基于树模型特征选择\n",
    "树模型基于分裂评价标准所计算的总的评分作为依据进行相关排序，然后进行特征筛选"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据未特征筛选维度 (2000, 229)\n",
      "训练数据特征筛选维度后 (2000, 71)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "clf = ExtraTreesClassifier(n_estimators=50)\n",
    "clf = clf.fit(train, target)\n",
    "\n",
    "model = SelectFromModel(clf, prefit=True)\n",
    "train_sel = model.transform(train)\n",
    "test_sel = model.transform(test)\n",
    "print('训练数据未特征筛选维度', train.shape)\n",
    "print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 树特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.09210871, 0.00578114, 0.00388741, 0.0047027 , 0.00324662,\n",
       "       0.00409547, 0.00560588, 0.00399393, 0.00499705, 0.00233944])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.feature_importances_[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features_import = pd.DataFrame()\n",
    "df_features_import['features_import'] = clf.feature_importances_\n",
    "df_features_import['features_name'] = features_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>features_import</th>\n",
       "      <th>features_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.092109</td>\n",
       "      <td>merchant_id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>228</th>\n",
       "      <td>0.085244</td>\n",
       "      <td>xgb_clf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>0.056583</td>\n",
       "      <td>lgb_clf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>0.007003</td>\n",
       "      <td>embeeding_72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>0.006930</td>\n",
       "      <td>embeeding_52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.006444</td>\n",
       "      <td>seller_most_1_cnt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207</th>\n",
       "      <td>0.006367</td>\n",
       "      <td>embeeding_80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>0.006110</td>\n",
       "      <td>embeeding_66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>190</th>\n",
       "      <td>0.006107</td>\n",
       "      <td>embeeding_63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>0.006077</td>\n",
       "      <td>embeeding_5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>0.005996</td>\n",
       "      <td>embeeding_17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>0.005913</td>\n",
       "      <td>embeeding_19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.005781</td>\n",
       "      <td>age_range</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>0.005715</td>\n",
       "      <td>embeeding_31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191</th>\n",
       "      <td>0.005701</td>\n",
       "      <td>embeeding_64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>0.005673</td>\n",
       "      <td>embeeding_38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.005648</td>\n",
       "      <td>cat_most_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.005606</td>\n",
       "      <td>brand_nunique</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.005488</td>\n",
       "      <td>user_cnt_0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>0.005485</td>\n",
       "      <td>embeeding_93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>166</th>\n",
       "      <td>0.005473</td>\n",
       "      <td>embeeding_39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>0.005472</td>\n",
       "      <td>tfidf_60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>0.005463</td>\n",
       "      <td>embeeding_0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>0.005427</td>\n",
       "      <td>embeeding_69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>205</th>\n",
       "      <td>0.005407</td>\n",
       "      <td>embeeding_78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>0.005402</td>\n",
       "      <td>embeeding_20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>0.005347</td>\n",
       "      <td>embeeding_36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>0.005328</td>\n",
       "      <td>embeeding_65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>0.005280</td>\n",
       "      <td>embeeding_42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>0.005277</td>\n",
       "      <td>tfidf_23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     features_import      features_name\n",
       "0           0.092109        merchant_id\n",
       "228         0.085244            xgb_clf\n",
       "227         0.056583            lgb_clf\n",
       "199         0.007003       embeeding_72\n",
       "179         0.006930       embeeding_52\n",
       "18          0.006444  seller_most_1_cnt\n",
       "207         0.006367       embeeding_80\n",
       "193         0.006110       embeeding_66\n",
       "190         0.006107       embeeding_63\n",
       "132         0.006077        embeeding_5\n",
       "144         0.005996       embeeding_17\n",
       "146         0.005913       embeeding_19\n",
       "1           0.005781          age_range\n",
       "158         0.005715       embeeding_31\n",
       "191         0.005701       embeeding_64\n",
       "165         0.005673       embeeding_38\n",
       "15          0.005648         cat_most_1\n",
       "6           0.005606      brand_nunique\n",
       "22          0.005488         user_cnt_0\n",
       "220         0.005485       embeeding_93\n",
       "166         0.005473       embeeding_39\n",
       "87          0.005472           tfidf_60\n",
       "127         0.005463        embeeding_0\n",
       "196         0.005427       embeeding_69\n",
       "205         0.005407       embeeding_78\n",
       "147         0.005402       embeeding_20\n",
       "163         0.005347       embeeding_36\n",
       "192         0.005328       embeeding_65\n",
       "169         0.005280       embeeding_42\n",
       "50          0.005277           tfidf_23"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_features_import.sort_values(['features_import'],ascending=0).head(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# features_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择前后区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Select Accuracy: 0.93 (+/- 0.00)\n",
      "Features Select Accuracy: 0.93 (+/- 0.00)\n"
     ]
    }
   ],
   "source": [
    "feature_selection(train, train_sel, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lgb特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: colsample_bylevel\n",
      "[LightGBM] [Warning] Unknown parameter: tree_method\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: colsample_bylevel\n",
      "[LightGBM] [Warning] Unknown parameter: tree_method\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.006242 seconds.\n",
      "You can set `force_row_wise=true` to remove the overhead.\n",
      "And if memory is not enough, you can set `force_col_wise=true`.\n",
      "[LightGBM] [Info] Total Bins 32114\n",
      "[LightGBM] [Info] Number of data points in the train set: 1200, number of used features: 224\n",
      "[LightGBM] [Warning] Unknown parameter: colsample_bylevel\n",
      "[LightGBM] [Warning] Unknown parameter: tree_method\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Start training from score -0.068100\n",
      "[LightGBM] [Info] Start training from score -2.720629\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[1]\tvalid_0's multi_logloss: 0.256738\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[2]\tvalid_0's multi_logloss: 0.256574\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[3]\tvalid_0's multi_logloss: 0.256518\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[4]\tvalid_0's multi_logloss: 0.25657\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[5]\tvalid_0's multi_logloss: 0.256756\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[6]\tvalid_0's multi_logloss: 0.25682\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[7]\tvalid_0's multi_logloss: 0.256989\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[8]\tvalid_0's multi_logloss: 0.257236\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[9]\tvalid_0's multi_logloss: 0.25712\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[10]\tvalid_0's multi_logloss: 0.257011\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[11]\tvalid_0's multi_logloss: 0.257042\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[12]\tvalid_0's multi_logloss: 0.257402\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[13]\tvalid_0's multi_logloss: 0.257419\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[14]\tvalid_0's multi_logloss: 0.257646\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[15]\tvalid_0's multi_logloss: 0.257552\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[16]\tvalid_0's multi_logloss: 0.257604\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[17]\tvalid_0's multi_logloss: 0.257797\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[18]\tvalid_0's multi_logloss: 0.257928\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[19]\tvalid_0's multi_logloss: 0.258142\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[20]\tvalid_0's multi_logloss: 0.25847\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[21]\tvalid_0's multi_logloss: 0.258654\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[22]\tvalid_0's multi_logloss: 0.258846\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[23]\tvalid_0's multi_logloss: 0.258962\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[24]\tvalid_0's multi_logloss: 0.258991\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[25]\tvalid_0's multi_logloss: 0.259334\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[26]\tvalid_0's multi_logloss: 0.259433\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[27]\tvalid_0's multi_logloss: 0.259912\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[28]\tvalid_0's multi_logloss: 0.260153\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[29]\tvalid_0's multi_logloss: 0.260576\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[30]\tvalid_0's multi_logloss: 0.26094\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[31]\tvalid_0's multi_logloss: 0.261198\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[32]\tvalid_0's multi_logloss: 0.26141\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[33]\tvalid_0's multi_logloss: 0.261614\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[34]\tvalid_0's multi_logloss: 0.261801\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[35]\tvalid_0's multi_logloss: 0.261931\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[36]\tvalid_0's multi_logloss: 0.262242\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[37]\tvalid_0's multi_logloss: 0.262492\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[38]\tvalid_0's multi_logloss: 0.26273\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[39]\tvalid_0's multi_logloss: 0.262855\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[40]\tvalid_0's multi_logloss: 0.263225\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[41]\tvalid_0's multi_logloss: 0.263311\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[42]\tvalid_0's multi_logloss: 0.263612\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[43]\tvalid_0's multi_logloss: 0.263937\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[44]\tvalid_0's multi_logloss: 0.264398\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[45]\tvalid_0's multi_logloss: 0.264822\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[46]\tvalid_0's multi_logloss: 0.264977\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[47]\tvalid_0's multi_logloss: 0.265401\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[48]\tvalid_0's multi_logloss: 0.265718\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[49]\tvalid_0's multi_logloss: 0.265859\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[50]\tvalid_0's multi_logloss: 0.266173\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[51]\tvalid_0's multi_logloss: 0.266544\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[52]\tvalid_0's multi_logloss: 0.266719\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[53]\tvalid_0's multi_logloss: 0.266817\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[54]\tvalid_0's multi_logloss: 0.267013\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[55]\tvalid_0's multi_logloss: 0.267385\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[56]\tvalid_0's multi_logloss: 0.267389\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[57]\tvalid_0's multi_logloss: 0.267662\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[58]\tvalid_0's multi_logloss: 0.267792\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[59]\tvalid_0's multi_logloss: 0.268017\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[60]\tvalid_0's multi_logloss: 0.268158\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[61]\tvalid_0's multi_logloss: 0.268437\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[62]\tvalid_0's multi_logloss: 0.268773\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[63]\tvalid_0's multi_logloss: 0.268824\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[64]\tvalid_0's multi_logloss: 0.269138\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[65]\tvalid_0's multi_logloss: 0.269357\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[66]\tvalid_0's multi_logloss: 0.269572\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[67]\tvalid_0's multi_logloss: 0.269786\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[68]\tvalid_0's multi_logloss: 0.270102\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[69]\tvalid_0's multi_logloss: 0.270435\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[70]\tvalid_0's multi_logloss: 0.270566\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[71]\tvalid_0's multi_logloss: 0.270679\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[72]\tvalid_0's multi_logloss: 0.271056\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[73]\tvalid_0's multi_logloss: 0.271474\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[74]\tvalid_0's multi_logloss: 0.27168\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[75]\tvalid_0's multi_logloss: 0.271918\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[76]\tvalid_0's multi_logloss: 0.271937\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[77]\tvalid_0's multi_logloss: 0.272113\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[78]\tvalid_0's multi_logloss: 0.27242\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[79]\tvalid_0's multi_logloss: 0.272712\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[80]\tvalid_0's multi_logloss: 0.27267\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[81]\tvalid_0's multi_logloss: 0.273019\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[82]\tvalid_0's multi_logloss: 0.272981\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[83]\tvalid_0's multi_logloss: 0.273218\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[84]\tvalid_0's multi_logloss: 0.27353\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[85]\tvalid_0's multi_logloss: 0.273649\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[86]\tvalid_0's multi_logloss: 0.273775\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[87]\tvalid_0's multi_logloss: 0.273835\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[88]\tvalid_0's multi_logloss: 0.274091\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[89]\tvalid_0's multi_logloss: 0.274422\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[90]\tvalid_0's multi_logloss: 0.274716\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[91]\tvalid_0's multi_logloss: 0.275082\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[92]\tvalid_0's multi_logloss: 0.275278\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[93]\tvalid_0's multi_logloss: 0.275447\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[94]\tvalid_0's multi_logloss: 0.275438\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[95]\tvalid_0's multi_logloss: 0.275778\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[96]\tvalid_0's multi_logloss: 0.27591\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[97]\tvalid_0's multi_logloss: 0.276129\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[98]\tvalid_0's multi_logloss: 0.276326\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[99]\tvalid_0's multi_logloss: 0.276449\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[100]\tvalid_0's multi_logloss: 0.276745\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[101]\tvalid_0's multi_logloss: 0.276895\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[102]\tvalid_0's multi_logloss: 0.276914\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n",
      "[103]\tvalid_0's multi_logloss: 0.277281\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's multi_logloss: 0.256518\n"
     ]
    }
   ],
   "source": [
    "import lightgbm\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.4, random_state=0)\n",
    "\n",
    "clf = lightgbm\n",
    "\n",
    "train_matrix = clf.Dataset(X_train, label=y_train)\n",
    "test_matrix = clf.Dataset(X_test, label=y_test)\n",
    "params = {\n",
    "          'boosting_type': 'gbdt',\n",
    "          #'boosting_type': 'dart',\n",
    "          'objective': 'multiclass',\n",
    "          'metric': 'multi_logloss',\n",
    "          'min_child_weight': 1.5,\n",
    "          'num_leaves': 2**5,\n",
    "          'lambda_l2': 10,\n",
    "          'subsample': 0.7,\n",
    "          'colsample_bytree': 0.7,\n",
    "          'colsample_bylevel': 0.7,\n",
    "          'learning_rate': 0.03,\n",
    "          'tree_method': 'exact',\n",
    "          'seed': 2017,\n",
    "          \"num_class\": 2,\n",
    "          'silent': True,\n",
    "          }\n",
    "num_round = 10000\n",
    "early_stopping_rounds = 100\n",
    "model = clf.train(params, \n",
    "                  train_matrix,\n",
    "                  num_round,\n",
    "                  valid_sets=test_matrix,\n",
    "                  early_stopping_rounds=early_stopping_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lgb_transform(train, test, model, topK):\n",
    "    train_df = pd.DataFrame(train)\n",
    "    train_df.columns = range(train.shape[1])\n",
    "    \n",
    "    test_df = pd.DataFrame(test)\n",
    "    test_df.columns = range(test.shape[1])\n",
    "    \n",
    "    features_import = pd.DataFrame()\n",
    "    features_import['importance'] = model.feature_importance()\n",
    "    features_import['col'] = range(train.shape[1])\n",
    "    \n",
    "    features_import = features_import.sort_values(['importance'],ascending=0).head(topK)\n",
    "    sel_col = list(features_import.col)\n",
    "    \n",
    "    train_sel = train_df[sel_col]\n",
    "    test_sel = test_df[sel_col]\n",
    "    return train_sel, test_sel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据未特征筛选维度 (2000, 229)\n",
      "训练数据特征筛选维度后 (2000, 20)\n"
     ]
    }
   ],
   "source": [
    "train_sel, test_sel = lgb_transform(train, test, model, 20)\n",
    "print('训练数据未特征筛选维度', train.shape)\n",
    "print('训练数据特征筛选维度后', train_sel.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### lgb特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 3, 0, 0, 0, 1, 1, 0, 1, 0])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.feature_importance()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sorted(model.feature_importance(),reverse=True)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择前后区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Select Accuracy: 0.93 (+/- 0.00)\n",
      "Features Select Accuracy: 0.93 (+/- 0.00)\n"
     ]
    }
   ],
   "source": [
    "feature_selection(train, train_sel, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
